A Real-Time Emotion Recognition System Using Keyboard, Mouse and Touchscreen Dynamics with Machine Learning

Abstract

This paper presents an enhanced framework for emotion recognition using Keystroke, Mouse, and Touchscreen (KMT) dynamics integrated with Artificial Intelligence and Machine Learning (AIML) techniques. The proposed system extends the study of Yang and Qin (2025), which identified key KMT features qualitatively, by introducing real-time data modeling and additional behavioral features. New parameters such as touch area, finger offset distance, swipe acceleration, and inter-key latency are combined with conventional features including typing speed and pressure intensity. User interaction data are collected through Python-based desktop and Android applications under different emotional states. Extracted features are processed and classified using supervised learning algorithms such as Random Forest and Support Vector Machine (SVM). Experimental results indicate that the inclusion of spatiotemporal and pressure-based features improves prediction accuracy and model robustness. The proposed framework supports personalized and adaptive humancomputer interaction systems.

Introduction

This paper presents a privacy-preserving emotion recognition system that uses Keystroke, Mouse, and Touchscreen (KMT) dynamics instead of cameras, microphones, or wearable sensors. Traditional emotion recognition methods often require specialized hardware and raise privacy concerns, whereas behavioral biometrics provide a low-cost and non-intrusive alternative.

The proposed framework collects user interaction data such as typing behavior, mouse movements, clicks, touchscreen taps, swipes, gesture duration, touch area, tap pressure, swipe acceleration, finger offset distance, and inter-key latency. After preprocessing and feature extraction, the system uses Machine Learning models, specifically Random Forest and Support Vector Machine (SVM), to classify emotions such as happy, calm, sad, stressed, anger, fear, disgust, and surprise.

The system was implemented using Python for Windows applications and Flutter/Dart for Android devices. Data were stored locally in CSV files, and all processing was performed on-device, ensuring user privacy and reducing deployment costs.

Experimental evaluation used an 80:20 train-test split and assessed performance using accuracy, precision, recall, F1-score, and confusion matrices. Results showed that Random Forest outperformed SVM, achieving 91.2% accuracy, 90.8% precision, 90.5% recall, and 90.6% F1-score, while SVM achieved 86.4% accuracy. The confusion matrix indicated only minor misclassification between similar emotions such as sadness and stress.

The findings demonstrate that combining multiple behavioral modalities significantly improves emotion recognition compared to single-input approaches. The framework provides a real-time, privacy-preserving, low-cost, and scalable solution suitable for applications in smart education, healthcare monitoring, workplace productivity, gaming, customer support, mobile applications, and adaptive human-computer interaction systems.

Conclusion

This paper presented a real-time emotion recognition system based on Keystroke, Mouse, and Touchscreen (KMT) dynamics integrated with machine learning techniques. The proposed framework provides a privacy-preserving and lowcost alternative to conventional camera-, speech-, and sensorbased emotion detection systems. Real interaction data were successfully collected through developed Windows and Android applications, and meaningful behavioral features were extracted for model training. Supervised learning algorithms such as Support Vector Machine (SVM) and Random Forest were applied for multi-class emotion classification. Experimental results demonstrated that combining multiple interaction modalities and enhanced features improved prediction accuracy, robustness, and real-time usability. The developed system was capable of generating mood predictions, reports, and structured datasets for future analysis. The major achievements of this work include successful real-time data capture, local machine learning prediction, cross-platform implementation, automated report generation, and practical deployment without additional hardware. The proposed framework can support future intelligent humancomputer interaction systems in education, healthcare, workplace analytics, and personalized computing environments.

References

[1] Y. Yang and Z. Qin, “Emotion recognition using keystroke, mouse, and touchscreen dynamics,” IEEE Access, vol. 13, pp. 12543–12558, 2025. [2] A. Kapoor and R. W. Picard, “Multimodal affect recognition in learning environments,” in Proc. IEEE Int. Conf. Multimedia and Expo, Toronto, ON, Canada, 2006, pp. 677–682. [3] S. D’Mello and J. Kory, “A review and meta-analysis of multimodal affect detection systems,” ACM Computing Surveys, vol. 47, no. 3, pp. 1–36, 2015. [4] F. Monrose and A. Rubin, “Keystroke dynamics as a biometric for authentication,” Future Generation Computer Systems, vol. 16, no. 4, pp. 351–359, 2000. [5] T. Ahmed and M. Traore, “Detecting emotion through mouse movement dynamics,” in Proc. Int. Conf. Human-Computer Interaction, Las Vegas, NV, USA, 2011, pp. 233–240. [6] J. Doe and A. Smith, “Touchscreen gesture analytics for user behavior modeling,” Journal of Mobile Computing, vol. 10, no. 2, pp. 88–97, 2022. [7] F. Pedregosa et al., “Scikit-learn: Machine learning in Python,” Journal of Machine Learning Research, vol. 12, pp. 2825–2830, 2011. [8] T. Chen and C. Guestrin, “XGBoost: A scalable tree boosting system,” in Proc. ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining, San Francisco, CA, USA, 2016, pp. 785–794. [9] J. Brownlee, “Support vector machines for classification,” Machine Learning Mastery, 2020. [Online]. Available: [10] https://machinelearningmastery.com/ [11] Python Software Foundation, “Python language reference, version 3.x,” 2025. [Online]. Available: https://www.python.org/

Copyright

Copyright © 2026 Fareed Babu Sayyed, Mr. D. Sunil Kumar, Mounika Namburu, Ande Uma Maheswari, Nookaraju Bodapati. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.